Skip to content

Add CrowS-Pairs task#70

Open
oskarvanderwal wants to merge 39 commits intobigscience-workshop:mainfrom
oskarvanderwal:main
Open

Add CrowS-Pairs task#70
oskarvanderwal wants to merge 39 commits intobigscience-workshop:mainfrom
oskarvanderwal:main

Conversation

@oskarvanderwal
Copy link

@oskarvanderwal oskarvanderwal commented Nov 10, 2021 •
edited
Loading

  • Evaluated on: GPT-2
  • Time evaluating on GPU: 00:48

Here is my attempt at implementing CrowS-Pairs and making it suitable for autoregressive models (closes #37).
Originally, CrowS-Pairs is designed for Masked Language Models, so I had to adapt their sentence scoring function (based on masking tokens). I am using perplexity instead to compare the sentences.

I have tested the task on GPT-2, and get the following results:

{
    "crowspairs_bias": 0.593501326259947,
    "crowspairs_bias_age": 0.5287356321839081,
    "crowspairs_bias_disability": 0.6,
    "crowspairs_bias_gender": 0.583969465648855,
    "crowspairs_bias_nationality": 0.44654088050314467,
    "crowspairs_bias_physical-appearance": 0.6349206349206349,
    "crowspairs_bias_race-color": 0.5775193798449613,
    "crowspairs_bias_religion": 0.6761904761904762,
    "crowspairs_bias_sexual-orientation": 0.7738095238095238,
    "crowspairs_bias_socioeconomic": 0.6686046511627907
}

oskarvanderwal and others added 30 commits October 8, 2021 12:49
@oskarvanderwal
Add template for CrowS-Pairs task
98f0beb
@oskarvanderwal
Adjust filename
a251d23
@oskarvanderwal
Loading CrowS-Pairs from url in right format
5d1a11d
@oskarvanderwal
Tokenize dataset sentences+plan for scoring func.
b8c3cd5
@oskarvanderwal
Add link to implementation scores crows-pairs
8bbbd7e
@oskarvanderwal
Implemented most of the eval logic (untested)
03ee5ee
@oskarvanderwal
Add scoring sentence as separate function
83d1cef
@oskarvanderwal
Remove unnecessary todo item
e0e7fbc
@oskarvanderwal
Add comment about dataset on huggingface datasets
4096287
@oskarvanderwal
Update crowspairs.py
1dce412 
Add dummy scoring function
@oskarvanderwal
Update crowspairs.py
aaf2bc1 
No more bugs, but haven't validated if it is implemented correctly yet.
Also need to fix score per bias type
@ozzypossie
Simplified the evaluation logic
eeabdc8
@ozzypossie
Remove print statement
d247e98
@ozzypossie
Test
b4bb270
@ozzypossie
Assume not start and stop token
6a664bb
@ozzypossie
Remove pair-score metric
43d50ae
@ozzypossie
Small bug fix
801dd06
@ozzypossie
Remove print statement
31c941c
@ozzypossie
Add confidence metric
a05b622
@ozzypossie
Add average model confidence to metric
35a9e18
@ozzypossie
Typo
fcdd28e
@ozzypossie
Comment out confidence metric
3d8416f
@oskarvanderwal
Change sentence score to perplexity
919dc1f
@oskarvanderwal
Update crowspairs.py
7df4df2
@oskarvanderwal
Update crowspairs.py
01db11d
@oskarvanderwal
Update crowspairs.py
c0a9fea
@oskarvanderwal
Update crowspairs.py
d1d50a4
@oskarvanderwal
Update crowspairs.py
ecfa84a
@oskarvanderwal
Update crowspairs.py
c979e93
@oskarvanderwal
Update crowspairs.py
0521b24
@oskarvanderwal oskarvanderwal mentioned this pull request Apr 28, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer

@sebastianGehrmann sebastianGehrmann sebastianGehrmann approved these changes

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add CrowS-Pairs to Full Benchmark

3 participants

@oskarvanderwal @sebastianGehrmann @ozzypossie